Fix staging embedding CI failures: stale-only self-healing fixture + org cleanup workflow#2068
Open
gramos-lb wants to merge 2 commits into
Open
Fix staging embedding CI failures: stale-only self-healing fixture + org cleanup workflow#2068gramos-lb wants to merge 2 commits into
gramos-lb wants to merge 2 commits into
Conversation
Co-authored-by: Cursor <cursoragent@cursor.com>
gramos-lb
requested review from
ChuckTerry,
KeshavSahoo,
carl-l-chua,
cyrusj89,
dsinha244,
gmadaan-hue and
snmodi21
July 21, 2026 23:15
CI runs pytest as a console script, which never puts the project directory on sys.path, so the new 'from tests.embedding_cleanup import ...' failed at conftest load (ModuleNotFoundError: No module named 'tests'). Local runs masked this because 'python -m pytest' adds the cwd. Insert the project directory in conftest before the tests.* import; verified against all three lanes with the console-script invocation.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fix the staging CI embedding failures: self-healing fixture + org cleanup workflow
What was broken
The staging org that all SDK CI runs share (
...G7FMH) has a hard cap of 10 custom embeddings per organization, and it's currently pinned at that cap by leaked test embeddings. Theembeddingfixture createssdk-int-<uuid>embeddings and deletes them in teardown — but cancelled/killed runs (the develop workflow hascancel-in-progress: true) skip teardown and leak them. Since the publicEmbeddingmodel has nocreated_at, a leaked embedding is indistinguishable from a live one, and the org never recovers. Result: every embedding fixture errors withMax limit of custom embeddings reached : 10 per organization, on develop and on every PR.Why the naive fix (delete everything named
sdk-int-*when the cap is hit) is wrong: hitting the cap doesn't mean the ten occupants are leaked. Up to 10 matrix jobs run concurrently against this org (5 python versions × integration/data, each with-n 32), so some occupants can be live fixtures another job is mid-test with — andlbox-alignerr's conftest uses the same name prefix. A blind sweep would delete embeddings out from under running tests.What changed
Timestamped names + stale-only self-healing (
tests/embedding_cleanup.py, fixture intests/conftest.py):sdk-int-ci-v2-<epoch>-<uuid>— the creation time rides in the name, which is the only way to establish staleness.LabelboxErrorre-raises immediately), the fixture deletes only custom embeddings in the v2 format older than 12 hours, then retries — up to 3 attempts with jitter. The TTL exceeds GitHub's 6-hour job ceiling, so a live fixture can never be classified stale. Legacy-format and foreign names are never touched automatically.One-time bootstrap (
.github/workflows/staging-embedding-cleanup.yml+tests/scripts/cleanup_staging_embeddings.py):workflow_dispatchonly,dry_rundefaults to true. Targets legacysdk-int-<32hex>leaks (which the automated path deliberately won't touch) plus stale-v2, using the same imported constants — no duplicated regex/TTL.api.lb-stage.xyz(checked at construction and again before listing); always prints the full candidate list before acting; real runs exit nonzero if any deletion fails.Unit coverage: 17 new tests in
tests/unit/test_embedding_cleanup.py— name parse/build rejection cases, stale selection, exact call-count oracles for the retry orchestration (no 4th attempt, no side effects on unrelated errors), endpoint wiring, fail-closed host check, dry-run and failure-exit behavior.Important: this PR's own embedding jobs will still be red
A new workflow can't be dispatched until it exists on the default branch, so the bootstrap can only run after merge — the legacy leaks keep the org pinned until then. Red embedding jobs on this PR are expected and are exactly the pre-existing failures this PR fixes.
Post-merge runbook
dry_run: true; review the candidate list.dry_run: false; a green run means the org is actually clean (any failed deletion exits nonzero).Residual:
lbox-alignerrstill creates legacy-format names, so a cancelled LBox Develop run can re-pin the org; the fix is re-running this workflow (migrating alignerr to v2 names is a sensible follow-up, deliberately out of scope here).Note on the second commit
The first push failed CI at collection: CI invokes pytest as a console script (via rye), which — unlike
python -m pytest— doesn't put the project directory onsys.path, so the newtests.*import in conftest wasn't resolvable (local validation usedpython -m, which masked it). Fixed with a guardedsys.pathbootstrap intests/conftest.pyright before that import; deliberately not via[tool.pytest.ini_options] pythonpath, which changes pytest's rootdir semantics and breaks the existingfrom ..conftestrelative imports (verified against clean develop).How to test
pytest tests/unit/test_embedding_cleanup.py(17 tests) or the full unit lane (282 passing).python -m tests.scripts.cleanup_staging_embeddings --helpfromlibs/labelbox.